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Abstract 

Fermion mixing is conveniently described using the effective Lagrangian 
formalism. We apply this approach to study top mixing in models with 
an infinite tower of Kaluza-Klein fermion excitations. In the Randall- 
Sundrum background with a boundary Higgs and phenomenologically vi- 
able values of the model parameters, the only effect eventually observable 
is the loss of universality of the top couplings. Their deviation from the 
SM predictions can be up to 4% if the five-dimensional Yukawa couplings 
times the AdSs curvature scale are < 10. 



1 Introduction 

Fermion mixing has been historically a window to new physics and a guide 
for model building. A primer example was the prediction of the cxistance of 
the charm quark to explain the suppression of strangeness-changing neutral 
currents (GIM mechanism) JjJ. Similarly the naturalness of such a suppression 
for all Flavour-Changing Neutral Currents (FCNC) implies that the isospin 
properties of the three standard families are the same J^]. However, this may 
be not what is required to describe the top quark. Its mixing is poorly known 
experimentally, leaving enough room for relatively large deviations from the 
Standard Model (SM) predictions (see Ref. || for a review). As a matter of 
fact, there are simple SM extensions where FCNC not involving the top quark 
are naturally suppressed in a large region of parameter space, and which allow 
for a large top mixing at the same time [Q. In the following, we first discuss 
the limits on top mixing || and the description of fermion mixing using the 
effective Lagrangian approach j^] . Then we apply the general results for heavy 
exotic fermions ]6[ to top mixing in the five-dimensional Randall-Sundrum (RS) 
model 0, g, |, [ft], |ll], Q and show that the size of possible SM deviations can 
be up to 4%. 



1 Presented by F.A. at NATO Advanced Study Institute 2000 "Recent Developments in 
Particle Physics and Cosmology", Cascais 26 June— 7 July 2000. 
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2 Top quark couplings 

The lowest dimension gauge interactions involving the top quark can be written 
in standard notation || O, |l4| 



L\ tq = -g s t^T a tG tia -~eijnA» (1) 

q—d,s,b 

t^(v? t -a? tl5 )tZ^ 



2 cos #w 
g 

2 cos #w 



E *7>f g -<75)<?^ + h. 



The first two terms are fixed by the unbroken gauge symmetry SU(3)c X U(1)q, 
while in the SM the charged currents are parametrized by , = , where 



Vt q is the Cabibbo-Kobayashi-Maskawa (CKM) matrix pa), and the neutral 
currents by vf t — \ — |sin 2 0^j aft = \ an< ^ v tq> a tq = 0- The SM Yukawa 
couplings are also diagonal. For discussing possible SM extensions using the 
effective Lagrangian formalism above the electroweak scale it is more convenient, 
however, to work with left (LH) and right (RH)-handed fields. Thus 

C\*< = -^(ulW^4 + u i R W^<)W+ + h.c. 

2cos0w V L ■ ij 7 Ul + UrX ^ 1 Ur 
-dlXffrdl - dk*«V4 - 2 sin2 °w Jem) ^ (2) 
where the summation on family indices i,j — 1, 2, 3 is understood and 

W _ W L +W R W _ W L -W R 

U 2 ' " — 2 ' 

„,z _ x L +x R 0c\n 2 a n „z _ x L -x R 
v — ^ z sm uwWi a — 2 ' 

with the electric charge Q term only substracted for the diagonal couplings. 

Present limits on FCNC involving the light quarks are somewhat stringent, 
but this is not yet the case for the top quark [|| 13, [T(|. For instance, from 



the size of K + — > Tt + vv and neglecting large cancellations with four-fcrmion 



contributions |-X"j''' R | < 4 x 10 5 , whereas available collider data only imply 



\X tq ' | < 0.66. Top couplings are constrained by electroweak data within the 
SM with a higher precision than will be by future direct measurements at large 
colliders. The question is if one can expect to observe departures from the SM 
top predictions in future experiments. The answer is positive as we will discuss 
in last section. 
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Table 1: Dimension 6 operators contributing to renormalizable Vtq couplings 
after SSB. 



<t>q 




O u<p = {4t<t>){qu$) 


O^w = 




= {^THD^iqi^q) 




0<f,B = 


\{<ft 4>)B l j iV B lxv 








OwB = 


-- (^t^)W^B^ 




= {tfiD^d^d) 










= (^ T eiD^)(u^d) 




h = 





3 Top quark effective Lagrangian 

The unbroken U(1)q protects the term proportional to J^ M in Eq. (||). The 
other Z and W couplings get contributions from higher orders in the effec- 
tive Lagrangian expansion after the Spontaneous Symmetry Breaking (SSB) of 
SU(2)l x U(1)y- The lowest order corrections result from the dimension 6 
Lagrangian Q 

Cf = ^O x +h.c, (4) 



where A is the effective high scale and the relevant operators O x are collected in 
Tablejj]. As a function of the coefficients a x the X and W coupling matrices in 
Eq. (||) read to order j^, with v the electroweak vacuum expectation value, || 

= ^-^(ag+a^-aJJ-agt)^, 
1 t> 2 

X ij R = - 2 ^2 + a \u)ij ' 

X dL t) 4. 1 ^ 4- ™ (1)t 4 ™ (3) 4- J 3)t \ 

X t* = + «!«,)«, (5) 

(2 

2 



2 A 2 1 



where 



V = V + ^(VA d L -AtV) (6) 

is a unitary matrix redefining the initial CKM matrix V to take into account the 
diagonalization of the quark mass matrices to order v 2 /A 2 . A u £ d only involve the 
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coefficients of the dimension 6 operators U tt>,d(t>, respectively. The non- linear 
realization of this Lagrangian is studied in Ref. p8[ . 

The couplings in Eq. (|^) incorporate features that are forbidden in the 
SM, namely, FCNC, RH neutral currents not proportional to J^ M , RH charged 
currents, and LH charged currents which are not described by a unitary matrix. 
These couplings can be directly determined for instance from processes involving 
the Vqq' vertices in which the final V is observed. For the top quark this 
will be possible at large colliders J|, [ljj. Trilinear couplings also contribute 
to four-fermion processes (such as mixing of neutral mesons), but four-fcrmion 
operators may contribute in this case as well. One can still use these processes 
to put limits on these couplings assuming that there are no strong cancellations 
between cubic and quartic couplings. 

The operators in the last column of Table |lj redefine the gauge bosons and 
thus the trilinear couplings in Eq. (||). However, they are flavour blind and 
hence also constrained by precise electroweak data involving only the light quark 
flavours, giving then unobservable corrections to top couplings. 



4 Bulk fermions in the RS model 

Models in extra dimensions can give towers of Kaluza-Klein (KK) excitations 
in four dimensions not far from the TeV scale . The integration of KK gauge 
bosons and/or fermions generate v 2 /A 2 corrections to the X and W couplings 
in Eq. (H) . In both cases they can be originated at tree level and can be a priori 
large |21| . However, the KK gauge boson contributions are proportional to their 
couplings to the SM fermions times their couplings to the SM gauge bosons, 
and the latter are typically - 0.01 @ ||. In this case the corresponding 
top corrections would be too small to be observable. In the following we will 
concentrate on the KK fermion contributions, studying in detail the case of bulk 
fermions in the RS model [Til] . 

In five dimensions there are no chiral fermions. Thus five-dimensional fermions 
\& are vector-like and can have a Dirac mass of the form 

£ D = -mi $ (f L $ R + § R * L ), (7) 

where ^> is the sum of the two four-dimensional "chiralities" ^l.r = ±75^l,r, 
transforming in the same way under the gauge group. The five-dimensional 
action containing the Yukawa interactions for the three standard families can 
be written in general with the Higgs on the TeV (irR) brane 

SVuk = -if d 4 x J dyV^ff Kj 5) 9i( x ^y)uj{x,y)4>(x) 

+ \f ) q i {x,y)d j {x,y)cj>{x)+Yi.c] 5{y-vR). (8) 
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Expanding the five-dimensional fields in KK towers and integrating over the 
fifth dimension, one obtains the four-dimensional mass Lagrangian which after 
SSB reads 



OO 



x u(nm) - (n)i ~ (m) j , \ d(nm) ~f(n)i "i(m)j 

A„> >u\ u y R + A„ v >d\ dL 



n , m — 
oo 

+£ 

n=l 



h.c. 



— (n)i (n)i 



u(n) (~{n)i ~(n)i -(n)i ~(n)i 



+M^ n >(u 



"R 



R 



+M„ 



\d L d^ R 



(9) 



where we have added to Eq. (||) the corresponding Dirac masses in Eq. (Q). 
The latter can always be taken diagonal. The four-dimensional masses can be 
written 



with 



i u . d 



x u,d(nrn) _ ,u,d ( n )i (m)j 
A ij ~ A ij a q u u,d > 

SM masses, 



(10) 



(n)i 



l u,d 



x u,d(5) h V 

V^kR ' 

■nkR/2 



(11) 



and 



250 GeV. 



(12) 



The factor e KkR in \^ d( - nm ) j s ^ ue ^ the rescaling of the boundary Higgs canon- 
ically normalized and the expansion coefficients /' n J give the n-th fermion wave 
functions at the ttR brane (k ~ 2.44 x 10 18 GeV and kR ~ 11 @). It must be 
noticed that odd fields are zero at the TeV boundary and then the odd chiral- 
ities {qR,UL, dh) have zero Yukawa couplings for a boundary Higgs. In matrix 
notation Eq. (ph reads 



_(0) 



7« 



a R 



A ij a q a « A ij a q a « 



a/; 








/\^jtLq iiu 



M 



9(1); 



(13) 
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and similarly for A4 d . It is finally convenient in order to compare with experi- 
ment, to rotate the zero modes 



U R — \ U RJV U R ' 



7(o)i 



,(0)i _ 



,'(0)J 



(14) 



where u R \ d' R , are the quark mass eigenstates up to mixing with the 
heavy KK excitations. In this basis the 3x3 light mass submatrices are 

(U^ikXtia^a^iU^ = V^mV, (U^) lk \ d kl a^ k af l {U d R ) lo = 

(15) 

with m"' the quark masses and V the CKM matrix in the absence of mixing. 



The effective Lagrangian resulting from integrating out the heavy vector-like 
KK fermions is explicitly given in Refs. H 0|. For the RS model the effective 
couplings in Eq. (||) read 

r ~, / 2 

l 



X 



uR 



{U U R) kj m u 3 , 



(Ul) kr V r )ml 



X 



dL 



<IR 



X 



wh = 



wl 




(U d ) kj m d , (16) 



(U q L )k 3 m d 



where at this order V can be replaced by V in X uR and W R . The extra 
contributions are products of 3 x 3 matrices, where the one in square brackets 
is diagonal and depends on the fermion location and then on the details of the 
model, and the others are unitary combinations of SM masses. The matrix in 
the middle can be further simplified noting that 

( a («))2 = ( a (D)2 = Xj (17) 



G 



which, up to a constant, leaves the diagonal elements as an infinite sum of 
the inverse of the KK heavy masses M^ n > squared. The lightest mass M^> 
plays the role of the effective scale in Eq. (||) , and the SM masses include the 
electroweak vacuum expectation value. The size of the corrections in the RS 
background depends on one mass parameter per flavour and 5J7(2)l x f (1)y 
multiplct. One can try to explain the observed pattern of fermion masses in 
this extended model j2j| or as we do in the following, simply ask how large can 
the SM corrections to the effective couplings be. 



5 Experimental implications 

In popular SM extensions the expected flavour changing branching ratios for the 
top quark are too small (< 10~ 4 ) to be observable at future colliders, except in 
the case of extra vector- like quarks near the electroweak scale (see Table ^ ||). 
For SM extensions with exotic quarks |24| electroweak data imply for instance 
< 0.08 and \X%\ < 0.16 j§, and then branching ratios <10~ 2 . The 
RS model which is a particular case with an infinite tower of vector-like quark 
singlets and doublets must satisfy these limits. In fact, the fit of Eq. ( [l6| ) 
to present data saturates these bounds. However for > 10 TeV [Q the 

corresponding Yukawa couplings X^k are ~ 100 and thus too large to recover 
the successful SM description below the TeV scale. Allowing only for X^k < 10 
the top coupling corrections are reduced, becoming only eventually observable 
for X t t and W^ b . In this region of parameters |AJfy-| ~ ™' ? " J AX t " L and then 
the main phcnomcnological constraints result from the loss of universality of the 
top couplings Jll| . Maximizing the top mixing in this region of parameter space 
and requiring at least the same % 2 as for the SM we have obtained departures 



Table 2: Experimental limits expected at LHC for the top quark flavour chang- 
ing branching ratios Br(t — > uZ,cZ) and values predicted in the SM, the two 
Higgs model (2H), supersymmetric models (SUSY) without R and with ft par- 
ity breaking and the SM extensions with exotic quarks. The branching ratio is 
defined as the decay rate divided by 1.56 GeV. 



LHC SM 2H SUSY Vector -like 

R, R quarks 

- 10" 4 ~ 10" ia ~ io- e - io- 4 ,io~ 8 - 10~ 2 
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from the SM top couplings of up to ~ 4%. Indeed for the Yukawa couplings 

/ 5.6 x 1CT 4 10 9.2 x 10" 3 
A^ (5) fc = -6.2 x 10- 4 -10 -9.2 x 10~ 3 | , (18) 
\ 1.2 x 10~ 3 -10 1.8 x 10~ 2 



-5.9xl0~ 5 -1.1 x 10~ 3 4.2 x 10~ 2 
A^ 5) fc = ( -7.7 x 10~ 5 5.2 x 10~ 4 -4.2 x 10~ 2 | (19) 
1.8 x 10~ 5 -1.5xl0~ 3 -4.6xl0~ 2 



and the coefficients 



a (o)«,c,t = Q.5603,0.1018,0.5603, 



,(0)i „(0)» 



</ 



0.5603, (20) 



where i = 1,2,3 is the family index, we find 

= 0.9608, Xg = 0.0013, (21) 

and 

/ 0.9752 0.2227 0.0036 \ 
\W£\ = 0.2227 0.9752 0.0402 . (22) 
\ 0.0096 0.0394 0.9804 / 

The corrections to the remaining X and W couplings are negligible. We have 
taken k = 2.44 x 10 18 and kR = 10.815 |p^| . The rotation matrices U entering 
in Eq. ( |l6| ) are fixed by the standard fermion masses and the CKM matrix 
through Eq. (||). Note that the value a (0) = 0.5603 in Eq. @ ensures that 
the bound on the lightest KK gauge boson mass Mf augc > 10 TeV is satisfied. 
This is needed to suppress the contribution of the tower of KK gauge bosons to 
electroweak observables below present experimental limits [ fTo] . 

In summary, we have shown that in the RS model there can be deviations 
from the SM Ztt and Wib couplings of at most ~ 4% and 2%, respectively, due 
to the top mixing with the KK fermion excitations. This has to be compared 
with the measurement of at LHC, for which an accuracy of 5% (10% for 
|W^| 2 in the cross section) is forseen as an ambitious but attainable goal ||. 
There are better prospects for the measurement of X^ t at TESLA. For instance, 
with a center of mass energy of 500 GeV, an integrated luminosity of 300 fb _1 
and unpolarized beams one expects to collect in the detector 34800 top pairs 
with one W decaying into ev or [iv and the other W decaying hadronically, 
reaching a precision of 2% in the determination of the Ztt coupling |25| . This 
precision will improve when all channels are included. 
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